Recommendations Based On Biometric Feedback From Wearable Device

ABSTRACT

Methods and architecture for using sensor data to offer content, such as purchase or viewing suggestions, to a viewer, are disclosed. Wearable or other external sensor devices may monitor a viewer while the viewer consumes a first media element. The sensor data may then be used to determine an emotional response of the viewer to the media element or to entities found within that element, such as actors, scenes, brands, or objects. Emotional response data for a user may be stored, and may be used either immediately or at a later time to deliver content or make content or purchase suggestions to the viewer.

FIELD

Aspects described herein generally relate to targeting media andpurchases to interested consumers. More specifically, aspects providefor detecting a variety of biological or behavioral parameters while aviewer consumes media and using that information to select additionalmedia which may be desired by the viewer.

BACKGROUND

The problem of targeting offerings to consumer taste is a difficult oneto solve in a way that maximizes consumer goodwill and minimizesmerchant expenses. Consumers generally want merchants or contentproviders to make accurate and helpful suggestions about what the viewermight like, but without inconveniencing the viewer by requiring theviewer to fill out extensive surveys or communicate at length with themerchant or content provider.

The ability to establish a profile of a viewer's habits or tastesbenefits advertisers, merchants, and consumers alike. Advertisersachieve a competitive advantage by targeting ads to favorable viewers,receiving a higher response rate and using expensive advertising spacemore efficiently. Merchants can suggest a purchase that a viewer mightneed without having realized the need, and increase their salessignificantly. The consumer benefits by being subjected to feweradvertisements that are irrelevant to the consumer, and possibly beingexposed to products that fit the consumer's tastes which the viewer wasnot aware of.

Existing methods for establishing a consumer taste profile focus on theconsumer's past actions and interactions with a provider. For example, aconsumer's purchase of an item may be used to suggest more items likethe purchased item. A consumer's rating of the purchased item through anonline review website or feedback form may be used to further adjust ataste profile positively or negatively for the item. A consumer whoviews one online video may be encouraged to watch a sequel after thevideo is complete, or another video in the same genre. A consumer whoviews a product's webpage may have the item associated with theconsumer's taste profile even if the viewer does not purchase the item,on the theory that the consumer is currently researching items of thegiven type and intends to buy one in the near future.

A limitation of these existing methods is that without the consumer'sactive participation in a rating system or survey, it is impossible totell whether the consumer enjoyed the purchase of the item, or theviewing of the video, or other use or consumption of media and products.

Providers may also attempt to proactively determine consumer tastes bysending surveys to consumers, interviewing consumers, or puttingtogether focus groups to gauge consumer reaction to media and brands.These polling methods may be very expensive, and risk destroyingconsumer goodwill by annoying consumers with unwanted communications anddemands on their time.

Thus, there is a need felt in the art for methods of accuratelydetermining the desires of a consumer in a passive manner, without theconsumer's conscious or active participation.

BRIEF SUMMARY

The following presents a simplified summary of various aspects describedherein. This summary is not an extensive overview, and is not intendedto identify key or critical elements or to delineate the scope of theclaims. The following summary merely presents some concepts in asimplified form as an introductory prelude to the more detaileddescription provided below.

To overcome limitations in the prior art described above, and toovercome other limitations that will be apparent upon reading andunderstanding the present specification, aspects described herein aredirected to using sensor readings from devices worn by a viewer, orotherwise monitoring a viewer, to determine the viewer's emotionalresponse to a media element. For example, the sensor readings and otherenvironmental information (e.g., time of day, temperature, etc.) can bemonitored and accumulated to build a viewer profile of emotionalresponses, e.g., favorable and unfavorable responses, to particularmedia, programs, movies, brands, products, persons, etc.

The sensor readings may be used to determine viewer heartbeat,breathing, posture, movement, speech, perspiration, facial expression,speech, tone, volume, and any other physical factors that may, alone orin conjunction with any of the physical characteristics above, change inresponse to a strong emotional reaction and thus be indicative of theviewer's underlying positive or negative emotional response to a mediaelement.

The viewer's emotional response, alone or in combination with the viewerprofile, may be used to suggest media content, particular portions orscenes within content, information, products, offers, and advertisementsto the viewer which are personally targeted and which do not requireadditional, and often undesirable, user inputs, such as requiring theuser to rate, comment, or reply to surveys.

Suggestions and offers may include offers to record streaming media, topurchase media, to license media, to change a currently displayed anddisfavored streaming media, to receive contextually relevant informationabout media, to purchase tickets to view a film, to purchasemerchandise, to receive coupons for merchandise, to take advantage oftemporarily deals for merchandise, or to select from amongalready-curated advertisements to view.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of aspects described herein and theadvantages thereof may be acquired by referring to the followingdescription in consideration of the accompanying drawings, in which likereference numbers indicate like features, and wherein:

FIG. 1 depicts a computing environment in which one or more aspectsdescribed herein may be implemented.

FIG. 2 depicts a system of connected computing devices, displays, andsensors which may be used to implement one or more aspects describedherein.

FIG. 3 depicts a method for determining an emotional response of aviewer based on observations of that viewer while consuming media,according to one or more aspects described herein.

FIG. 4 depicts an illustrative set of sensor and media consumption dataused to determine an emotional response of a viewer, according to one ormore aspects described herein.

FIG. 5 depicts an illustrative embodiment of using sensor data to offerrecording of media in which the viewer is interested.

FIG. 6 depicts an illustrative embodiment of using sensor data to offeralternative media to media in which the viewer is interested.

FIG. 7 depicts an illustrative embodiment of using sensor data to offercontextually relevant information about a media in which the viewer isinterested.

FIG. 8 depicts an illustrative embodiment of using sensor data todetermine viewer interest in a movie and offer to sell tickets to thatmovie to the viewer.

FIGS. 9A and 9B depict an illustrative embodiment of using sensor datato determine that a viewer has moved away from a display showing mediain which the viewer is interested and offering to continue display ofthe media on a new display for seamless viewing.

FIG. 10 depicts an illustrative embodiment of using sensor data todetermine viewer interest in merchandise and offer coupons or otherdeals to the viewer for the merchandise.

DETAILED DESCRIPTION

In the following description of the various embodiments, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration various embodiments in whichaspects described herein may be practiced. It is to be understood thatother embodiments may be utilized and structural and functionalmodifications may be made without departing from the scope of thedescribed aspects and embodiments. Aspects described herein are capableof other embodiments and of being practiced or being carried out invarious ways. Also, it is to be understood that the phraseology andterminology used herein are for the purpose of description and shouldnot be regarded as limiting. Rather, the phrases and terms used hereinare to be given their broadest interpretation and meaning. The use of“including” and “comprising” and variations thereof is meant toencompass the items listed thereafter and equivalents thereof as well asadditional items and equivalents thereof. The use of the terms“mounted,” “connected,” “coupled,” “positioned,” “engaged” and similarterms, is meant to include both direct and indirect mounting,connecting, coupling, positioning and engaging.

FIG. 1 depicts one example of a network architecture and data processingdevice that may be used to implement one or more illustrative aspectsdescribed herein. Various network nodes 103, 105, 107, and 109 may beinterconnected via a wide area network (WAN) 101, such as the Internet.Other networks may also or alternatively be used, including privateintranets, corporate networks, LANs, wireless networks, personalnetworks (PAN), and the like. Network 101 is for illustration purposesand may be replaced with fewer or additional computer networks. A localarea network (LAN) may have one or more of any known LAN topology andmay use one or more of a variety of different protocols, such asEthernet. Devices 103, 105, 107, 109 and other devices (not shown) maybe connected to one or more of the networks via twisted pair wires,coaxial cable, fiber optics, radio waves or other communication media.

The term “network” as used herein and depicted in the drawings refersnot only to systems in which remote storage devices are coupled togethervia one or more communication paths, but also to stand-alone devicesthat may be coupled, from time to time, to such systems that havestorage capability. Consequently, the term “network” includes not only a“physical network” but also a “content network,” which is comprised ofthe data—attributable to a single entity—which resides across allphysical networks.

The components may include data server 103, web server 105, and clientcomputers 107, 109. Data server 103 provides overall access, control andadministration of databases and control software for performing one ormore illustrative aspects described herein. Data server 103 may beconnected to web server 105 through which viewers interact with andobtain data as requested. Alternatively, data server 103 may act as aweb server itself and be directly connected to the Internet. Data server103 may be connected to web server 105 through the network 101 (e.g.,the Internet), via direct or indirect connection, or via some othernetwork. Viewers may interact with the data server 103 using remotecomputers 107, 109, e.g., using a web browser to connect to the dataserver 103 via one or more externally exposed web sites hosted by webserver 105. Client computers 107, 109 may be used in concert with dataserver 103 to access data stored therein, or may be used for otherpurposes. For example, from client device 107 a viewer may access webserver 105 using an Internet browser, as is known in the art, or byexecuting a software application that communicates with web server 105and/or data server 103 over a computer network (such as the Internet).

Servers and applications may be combined on the same physical machines,and retain separate virtual or logical addresses, or may reside onseparate physical machines. FIG. 1 depicts just one example of a networkarchitecture that may be used, and those of skill in the art willappreciate that the specific network architecture and data processingdevices used may vary, and are secondary to the functionality that theyprovide, as further described herein. For example, services provided byweb server 105 and data server 103 may be combined on a single server.

Each component 103, 105, 107, 109 may be any type of known computer,server, or data processing device. Data server 103, e.g., may include aprocessor 111 controlling overall operation of the rate server 103. Dataserver 103 may further include RAM 113, ROM 115, network interface 117,input/output interfaces 119 (e.g., keyboard, mouse, display, printer,etc.), and memory 121. I/O 119 may include a variety of interface unitsand drives for reading, writing, displaying, and/or printing data orfiles. Memory 121 may further store operating system software 123 forcontrolling overall operation of the data processing device 103, controllogic 125 for instructing data server 103 to perform aspects describedherein, and other application software 127 providing secondary, support,and/or other functionality which may or may not be used in conjunctionwith other aspects described herein. The control logic may also bereferred to herein as the data server software 125. Functionality of thedata server software may refer to operations or decisions madeautomatically based on rules coded into the control logic, made manuallyby a viewer providing input into the system, and/or a combination ofautomatic processing based on viewer input (e.g., queries, data updates,etc.).

Memory 121 may also store data used in performance of one or moreaspects described herein, including a first database 129 and a seconddatabase 131. In some embodiments, the first database may include thesecond database (e.g., as a separate table, report, etc.). That is, theinformation can be stored in a single database, or separated intodifferent logical, virtual, or physical databases, depending on systemdesign. Devices 105, 107, 109 may have similar or different architectureas described with respect to device 103. Those of skill in the art willappreciate that the functionality of data processing device 103 (ordevice 105, 107, 109) as described herein may be spread across multipledata processing devices, for example, to distribute processing loadacross multiple computers, to segregate transactions based on geographiclocation, viewer access level, quality of service (QoS), etc.

One or more aspects described herein may be embodied in computer-usableor readable data and/or computer-executable instructions, such as in oneor more program modules, executed by one or more computers or otherdevices as described herein. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data typeswhen executed by a processor in a computer or other device. The modulesmay be written in a source code programming language that issubsequently compiled for execution, or may be written in a scriptinglanguage such as (but not limited to) HTML or XML. The computerexecutable instructions may be stored on a computer readable medium suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects, and such data structures are contemplatedwithin the scope of computer executable instructions and computer-usabledata described herein.

FIG. 2 depicts a system of connected computing devices which may be usedto implement one or more aspects described herein.

A computing device, such as media server 200, may transmit various mediaelements through a network 205 for eventual consumption by a viewer.Media server 200 may be a web server sending webpages, images, streamingvideo, social media pages, or other files through network 205 to aviewer's computing device 215. Media server 200 may be a head endsending a video stream via cable to a set top box 240 in a viewer'shome, for display on television 210. Media server 200 may be a serverthat facilitates an online computer game and transmits data to theviewer in-game as the viewer plays on computing device 215 or gamingconsole 245.

Multiple devices may display media received from media server 200 to theviewer. Display 210 may be a television or other display connecteddirectly to network 205 or may be a television connected to a cablenetwork via set top box 208. Computing device 215 may be a desktopcomputer, laptop computer, or other computing device, and may beconnected to network 205 directly, or may be connected wirelessly viawireless access point 230. Portable computing device 220 may be a tabletcomputer, personal digital assistant (PDA), cellular phone, or smartphone, and may be connected to network 205 by wireless access point 230or by mobile broadband such as 3G, 4G, or LTE. Wireless access point 230may be a wireless router, hotspot, cell tower, repeater, or any otherdevice that facilitates a data connection.

Device 225 may be a smart phone, console game controller, televisionremote control, tablet computer, activity tracking device, sleeptracking device, wristband computing device, digital watch, or otherwearable device. It may be worn by, attached to, or kept in closeproximity to the viewer while the viewer is using any of devices 210,215, 220, or 245. It may have sensors including, but not limited to,multiple-axis accelerometers, gyroscope, thermometer, galvanometer,pressure sensor, mass spectrometer, microphone, camera, globalpositioning system (GPS), or motion sensor, or may be connected toperipheral devices with one or more such sensors. Devices 210, 215, 220,and 245 may also incorporate one or more of these sensors, and theviewer may also be observed by other sensors, such as cameras of a homesecurity system, or sensors in an automobile.

Sensors in devices 210, 215, 220, 225, and 245, and other sensors may beable to determine a variety of physical attributes of the viewer. Forexample, accelerometers, pressure sensors, or other sensors in a devicein contact with the viewer or worn by the viewer may be able to sensethe viewer's pulse and determine the viewer's heart rate.Accelerometers, pressure sensors, or other sensors may also be able todetermine the viewer's breathing rate, the depth or shallowness of theviewer's breaths, or sharp intakes or exhalations of breath. Agalvanometer or other sensor may be able to determine that the viewer isor is not perspiring. A thermometer, infrared camera, or other sensormay be able to detect minute changes in the viewer's core bodytemperature or in the temperature of particular parts of the body suchas the face. A mass spectrometer or other sensor may be able to detecthormones or pheromones released in or from the viewer's body.Accelerometers, cameras, motion sensors, pressure sensors, or othersensors may be able to detect body movements or gestures by the viewer.A pressure sensor or other sensor may be able to detect tension in theviewer's body, such as a clenched first or a contracted muscle. A cameraor other sensor may be able to detect the viewer's posture or facialexpression. A microphone or other sensor may be able to detect thevolume, pitch, tone, or content of the viewer's speech or otherverbalizations. A global positioning system (GPS) sensor may be used todetermine the viewer's location.

Devices 210, 215, 220, 225, and 245 may also be configured to receiveviewer input in the form of pressing a button, touching a touchscreen,using a stylus on a touchscreen, manipulating a switch or joystick,shaking a controller to activate an accelerometer, moving a mousecursor, clicking with a mouse, or control via voice commands.

Devices 210, 215, 220, 225, and 245 and any other sensors may usenetwork 205 directly or indirectly transmit sensor and input data, alongwith precise timestamps of when the data was recorded, to a computingdevice, such as a profile server 235. The computing device may store adatabase of viewer information, media information, and sensorinformation.

FIG. 3 depicts aspects of an embodiment. In step 300, the determinationmay be made that a viewer is consuming media on a display device. Thisdetermination may be made either by transmitting the media to the viewerdirectly, or by collaborating with a content provider that allows aninterface for querying the current media consumption behavior (such as acontent provider who shares viewer information with advertisers).

In step 305, the sensors in devices 210, 215, 220, 225, and 245 and anyother sensors may monitor sensor data at regular intervals, such asmultiple times per second, every second, or after a predetermined numberof seconds. The precise time of the sensor record may also be stored.Alternatively, the sensors may store sensor data only with there is achange in a sensor reading compared to a previous observation, ratherthan storing a same value at repeated intervals.

The sensors may also record corresponding environmental characteristics,such as time of day (as measured by an internal clock), light level (asmeasured by an optical sensor), temperature (as measured by athermometer), location (as measured by global positioning system (GPS)sensor), weather events (such as lightning, thunder, rain, hail, snow,tornado, or hurricane, as measured by cross-referencing a GPS readingwith a weather report database). Environmental characteristics may belater used to eliminate false positive or false negative readings, suchas a determination that a person was angry upon viewing media because ofincreased temperature being discounted if the person is in an area withabnormally high heat, or a determination that a person was fearful afterviewing media being discounted if there was a tornado warning in effectat the time.

The recorded sensor data and timestamps of the data may be transmittedto a computing device, such as a profile server 235. If analysis of rawsensor data is needed to understand its significance, such astransforming raw audio data into a transcript of spoken words, ortransforming raw accelerometer data into the gesture it signifies, theanalysis may be performed at the sensor before transmission, at thecomputing device itself, or at an intermediate device.

In step 310, computing device 235 may attempt to determine an emotionalresponse which the sensor data signifies at each recorded time.

For example, intense excitement and interest in what is being viewed maybe determined upon a sensor reading of physical attributes or behaviorshowing any of increased heart rate, holding breath, being silent,speaking rapidly, particular speech content and vocalizations (such as“Ooh!” or “Ah!” or “Neat!”), holding still, leaning forward or onto theedge of a seat, or a combination of these or of other factors known inthe psychological or medicinal arts to indicate excitement.

Happiness or amusement may be determined upon a sensor reading ofphysical attributes or behavior showing any of laughter, a smilingfacial expression, particular speech content that contains positivelyconnoted words, relaxation of muscles, or a combination of these or ofother factors known in the psychological or medicinal arts to indicatehappiness.

Fear may be determined upon a sensor reading of physical attributes orbehavior showing any of increased heart rate, shaking or trembling,sweating/perspiration, increased pitch of voice, particular speechcontent with key words such as “Scary!”, gasping, a facial expression offear, sensation of symptoms associated with adrenaline or a “fight orflight” reaction, sensation of fear-induced pheromones, or a combinationof these or of other factors known in the psychological or medicinalarts to indicate fear.

Disinterest or apathy may be determined upon a sensor reading ofphysical attributes or behavior showing any of restless movement,speaking in a normal voice, speaking over the sound of the media, abored tone, speech content with keywords like “bored”, lack of eyecontact with currently playing media, a relaxed/apathetic facialexpression sensed to correlate with boredom, or a combination of theseor of other factors known in the psychological or medicinal arts toindicate disinterest.

Anger or antipathy may be determined upon a sensor reading of physicalattributes or behavior showing any of increased body temperature,increased volume, speech content with negatively connoted or offensivekeywords, tone of voice, agitated movement, or tension in muscles, or acombination of these or of other factors known in the psychological ormedicinal arts to indicate anger.

Other emotions, such as sadness, disgust, affection, or suspicion, maybe determinable by facial expression, tone of voice, speech content andthe detection of keywords or sentiments therein, or other physicalattributes or behavior known in the psychological or medicinal arts toindicate an emotion.

Emotional responses may be determined either by a persistent sensorreading (such as continuing to determine a smiling facial expressionover a period of time) or may be determined only in response to a changein sensor reading (such that beginning to perspire indicates an emotion,but continuing to sense unevaporated perspiration does not). A change insensor readings may be deemed significant based on an absolute change(such as an increase in heart rate of 20 or more beats per minute), arelative change (such as a 15% or more increase in heart rate), or areference to a historical or typical value (elevated, but returning to“normal”).

In step 315, computing device 235 may obtain or retrieve the viewer'smedia consumption data. The computing device may transmit a request toanother computing device, such as media server 200, to share what mediathe viewer was consuming at each timestamp for which an emotional statehas been determined. For example, if other computing device 200 is a webserver, it may know which web page or other online content was sent tothe viewer's browser immediately before a given timestamp. If othercomputing device 200 is a head end, it may know what TV program, film,commercial advertisement, or other offering was being displayed to theviewer's television by a set top box. Other computing device 200 mayhave means for determining what programs the viewer is using on acomputer, what page of what e-book is being read by the viewer, or whatapps are being used on a smartphone by the viewer at a given time.

Additionally, other computing device 200 may have access to the mannerin which the viewer obtained access to the media the viewer was viewing.For example, a viewer who has paid for access to the mediaspecifically—such as by purchasing a cable television show “on demand”,purchasing a ticket to a movie theater, purchasing or renting videocontent through an online distributor, or ordering physical media froman online merchant—may be determined to have elevated interest in themedia or in actors or other elements featured in the media, and thiselevated interest can be combined with the biometric feedback data indetermining a positive emotional response to the media or those actorsor other elements featured in the media at the time of the biometricfeedback indicating positive emotional response. A viewer who hasincurred a cost other than directly paying for the media, such aschoosing to watch free media with interspersed mandatory advertisements,or paying for a subscription to a service with many available media forconsumption, may be determined to have an equally elevated interest asfor a direct purchase, a less elevated interest, or no additionalinterest at all compared to media for which the viewer incurred no cost.

In step 320, computing device 235 may take the compilations of vieweremotion at points in time and viewer media use at those same points intime to determine the viewer's emotional responses to the media elementthat the viewer was consuming at the time of the sensor reading. Thegranularity of the determination may be limited to a viewer reaction toan entire show or movie as a whole, or increased, such as by determiningthe viewer response to a particular scene in a program, or determiningviewer response to a particular brand or item in an advertisement.

In step 325, computing device 235 may, optionally, store thedetermination in a database with records that associate a viewer, amedia element, and an emotion by the viewer to the media element. Insome embodiments where the disclosed methods involve real timeprocessing, storing may not be necessary. The media elements for whichemotions are monitored, detected, analyzed and/or stored may be entireitems of content, such as shows or movies, or portions of content. Forexample, a viewer may have a stored association with an entiretelevision series, such as M*A*S*H, or with a particular episode, suchas the M*A*S*H series finale, or with a series of movies, such as theRocky series, or only a particular movie, such as Rocky IV. A viewer mayhave a stored positive association to a particular season of atelevision show, such as the first season of Arrow, while having astored negative association with another season of the same televisionshow, such as the fourth season of Arrow.

Media elements may also be or comprise specific items, brands, scenes,persons, or other elements which are found in content that is presentedwhen sensor readings are recorded. For example, a viewer may have astored emotional reaction to a character, like Darth Vader, and have aseparate emotional reaction for an actor who plays the character, likeJames Earl Jones. A viewer may have a stored emotional reaction to atype of vehicle, like an airplane or sports car, or a specificmanufacturer, such as Lockheed Martin or Porsche. A viewer may have apositive stored emotional reaction to a scene, such as the “I'll havewhat she's having” scene from When Harry Met Sally, while remainingneutral or even negative to the film as a whole, or to the two actorsinvolved, Billy Crystal and Meg Ryan. A viewer may have a stronglynegative stored emotional reaction to guns or gunfire regardless of thegenre or actors involved. A viewer may have a positive stored emotionalreaction to animals featured in a film, or to particular species orbreeds of animals. A viewer may have a positive stored emotionalreaction to a genre such as “western” or “murder mystery.” A viewer mayhave a positive stored emotional reaction to a particular food, such astacos, or to a restaurant brand, food brand, or beverage brand featuredin media content. These records may be compiled into a comprehensiveviewer profile that stores and tracks viewer emotional responses to awide variety of possible inputs.

In step 330, the stored records and/or viewer profile may be used tooffer a media element to the viewer. Selection of the media element maybe based on past positive reactions to that element, past positivereactions to similar elements, past negative reactions to otherelements, a currently sensed emotional state of the viewer, currentlyobserved behavior of the viewer, the location of the viewer, whichdisplays or devices the viewer is currently using, or any combination ofthese factors. Computing devices 200 or 235 may then transmit to theviewer a notification that the selected media is available forconsumption, purchase, or other use.

In an alternative embodiment, a group of viewers may be watching adisplay simultaneously, each of whom may be individually currentlysensed by a wearable device sensor or may have been individually sensedby a wearable device sensor in the past to establish an emotionalprofile on computing device 235. Computing devices 200 and/or 235 maydetermine that each member of the group is watching the same display,and the current information from the sensors and/or past informationfrom emotional profiles of multiple viewers may be combined to make arecommendation to the entire group and notify them that selected mediais available for consumption, purchase, or other use.

Selection for a group may be based on minimizing negative expectedreaction to the selection across the group, maximizing positive expectedreaction to the selection across the group, maximizing positive expectedreaction under the constraint that no viewer has an expected negativereaction, or another metric. Selection may also be based upon selectinga media containing elements each of which at least one viewer has anexpected positive reaction to. For example, if a first viewer has beendetermined to enjoy Sean Connery while a second viewer likes sportscars, a James Bond film containing both may be suggested. If the secondviewer likes the actor Alec Baldwin or likes sailing instead of sportscars, the film The Hunt For Red October might be suggested instead.

In the foregoing embodiments, the roles of the computing devices 200 and235 may be filled by two distinct devices, a single device, or amultitude of modules spread across many devices, transmitting mediadata, sensor data, analysis of sensor data, or other data amongthemselves.

FIG. 4 depicts an example set of sensor and media data of a viewer'sphysical attributes which might be captured over a period of time for aviewer of multiple display devices.

At time 400, the viewer is browsing a news site on smartphone display455 while Advertisement #1 begins playing on television display 450. Theviewer's heart rate 460 is normal, audio sensor data analyzed todetermine vocalizations 465 shows that the viewer is silent, andaccelerometer and camera data analyzed to determine movement 470 showsthe viewer is pacing around. Accordingly, computing device 235 maydetermine that the viewer was feeling disinterest at time 400, and wasunimpressed by the news site and by Advertisement #1.

At time 405, the viewer is determined to sit down, and heart rate beginsdramatically increasing. Microphone data is analyzed to determine thatthe viewer has shouted “Look at that!” Computing device 235 maydetermine that the viewer is now feeling extreme interest at time 405.By comparing with media information regarding displays 450 and 455,computing device 235 may determine that the viewer has an extremeinterest in either Online Shopping Item #1, which is displayed on thesmartphone, or an item in Advertisement #2, which is displayed on thetelevision. The determination between the two may be made on the basisof location (the viewer is sitting in a chair directly in front of thetelevision), movement (the viewer set down the smartphone before theexclamation), camera data (the viewer is making eye contact with thetelevision), or order of events (the viewer was viewing the online itemfor a period of time before time 405 without reaction, but reactedimmediately when the advertisement began at time 405). Computing device235 may then update the viewer profile to record the viewer's interestin the item or brand advertised in Advertisement #2.

At time 410, the viewer shifts around after having been still, and heartrate begins to return to normal. Although heart rate has been elevatedthe entire time that the viewer has used a smartphone to display OnlineShopping Item #2, the fact that heart rate did not increase during thetime frame may be used to determine a lack of interest in OnlineShopping Item #2. Likewise, the decreasing heart rate during viewing ofAdvertisement #3 and Online Shopping Item #3 may be used to determinedisinterest in products displayed therein.

At time 415, the viewer turns off display 455 and turns his attentionsolely to a program on display 450. Shortly after the program begin, theviewer's detected heart rate elevates, the viewer is detected to leanforward, and the viewer, who might have been involved in a conversation,stops talking abruptly. These signals may be combined to determine thatthe viewer is intensely interested in the program that the viewer iscurrently viewing. Computing device 235 may request data from a headend, set top box, streaming media server, or other media provider todetermine which program the viewer is watching. Once the program isidentified, computing device 235 may create an entry recording that theviewer is intensely interested in the particular program which theviewer was watching.

At time 420, the viewer's heart rate begins to slow, allowing computingdevice 235 to determine that the intensity of the viewer's emotionalreaction is decreasing. This determination may allow computing device235 to increase the granularity of the records regarding the viewer,potentially showing that the viewer was interested in the previous sceneat time 415 because it contained a particular actor, brand, or genrethat is not present in a scene at time 420. Computing device 235 maythen update its records to show interest by the viewer in both in theprogram and the actor, brand, or genre, or alternatively may replace therecord of the program with the actor, brand, or genre. Repeated sensordeterminations over many airings of a program may aid in determiningwhat element in particular is generating interest by the viewer.

At time 425, the viewer's sensor data has stabilized at a slightlyelevated heart rate and with no changes in posture or speech. Theslightly elevated heart rate may be considered as evidence by computingdevice 235 of general interest in the program as a whole, or may bediscounted as an artifact from the extreme increase in heart rate from aparticular scene.

At time 430, the viewer has reactivated display 455 and is using it tooperate a search engine. This information may be used by computingdevice 235 to determine that the viewer's interest in the program haswaned enough that the user is distracted by other devices.Alternatively, computing device 235 may be able to determine the queriesthat the viewer was entering into the search engine. Queries related tothe program on display 450 may be used to determine that the viewer isstill highly interested in the program despite presently payingattention to a different device.

At time 435, the program on display 450 is coming to a close, and theviewer has a spiking heart rate, is not focused on another display, andsays “Oooh!” These factors may again be used by computing device 235 todetermine intense interest in the program by the viewer, perhaps inresponse to a “cliffhanger” ending to the program. Although the user'sheart rate remains elevated for a subsequent time, computing device 235may determine that the viewer is not interested in the followingAdvertisement #4, discounting the heart rate as an artifact from theprogram, and considering the additional determination that the viewerstands up after time 435 and may not even be viewing display 450 afterstanding up.

FIGS. 5-10 illustrate a series of embodiments provided merely todemonstrate the variety of applications of the novel matter herein. Theyin no way circumscribe or limit the scope of the claims or of otherembodiments.

FIG. 5 depicts one embodiment, wherein physical characteristics of theviewer—such as viewer heartbeat, breathing, posture, movement, speech,perspiration, facial expression, speech, tone, and volume—may bemonitored by one or more wearable biometric sensors while watching astreaming media program 505, such as a wrestling match, on display 500.The sensors may send the sensor readings to computing device 235, whichmay cross-reference the sensor data with records from the viewer'stelevision provider, and may immediately determine that the viewer isexperiencing a positive reaction to the program, such as excitement orhappiness. Computing device 235 or 200 may determine that the viewerowns a device capable of recording input to display 500 or recordinginput to a set top box that controls the display. The display 500 maythen be caused to display a message 510 to the viewer such as “Press‘OK’ to save this program to your DVR.” The viewer might then be able topress a button on a remote control or set top box controller, which maybe configured to record the remainder or the entirety of the program tothe recording device for later display.

Alternatively, computing device 235 may determine that the viewer iscurrently holding a mobile device, and cause message 510 to be displayedon the mobile device display instead. Tapping or otherwise interactingwith the message may cause the mobile device to communicate the viewer'sselection to computing device 235, which may then cause the recordingdevice to record a portion of the program, the remainder of the program,or the entirety of the program. Tapping or otherwise interacting withthe message may cause the mobile device to communicate directly with theset top box or other recording device to cause recording of the program.

FIG. 6 depicts another embodiment, wherein the viewer may be monitoredby wearable biometric sensors while watching another streaming mediaprogram 605, such as a nature documentary, on display 600. The sensorsmay send sensor readings to computing device 235, which maycross-reference the sensor readings with records from the viewer'stelevision provider, internet service provider, cellular networkprovider, or the like, and may immediately determine that the viewer isexperiencing a negative reaction to program 605, such as boredom ordisgust.

Computing device 235 may then determine a list of program offeringscurrently available to the viewer, for example being broadcast forpickup by antenna, available under the viewer's television subscriptionplan, or freely available online. Computing device 235 may then selectan available program to offer instead. The selection may be based on theviewer's existing profile, selecting a program like one to which theviewer expressed a positive emotional reaction in the past, or may bebased on merely selecting a program to which no negative emotionalreaction has been expressed in the past. Computing device 235 may thencause display 600 to display a message 610 informing the viewer that apreferable program is available. The message may simply inform theviewer of the existence of the preferable program, or may allow theviewer to press a button or issue a command to immediately display thepreferable program instead of the current program.

Alternatively, computing device 235 may determine that the viewer iscurrently holding a mobile device, and cause message 610 to be displayedon the mobile device display instead. Tapping the message may cause themobile device to communicate the action directly to a set top box tochange the currently displayed program, or may communicate withcomputing device 235, which may directly or indirectly cause the displayto change the currently displayed program.

FIG. 7 depicts another embodiment, wherein the viewer may be monitoredby sensors while watching media element 705 on display 700. The sensorsmay send sensor readings to computing device 235, which maycross-reference the sensor readings with records from the viewer'stelevision provider, internet service provider, cellular networkprovider, or the like, and may immediately determine that the viewer isdemonstrating interest in media element 705. Computing device 235 maythen determine that there is contextually related information aboutmedia element 705 which the viewer may be interested in. Computingdevice 235 may then cause display 700 or mobile device 710 to displaymessage 715 with the contextually related information, or with a link tothe information.

If the media element is an advertisement, for example, the message 715may include a hyperlink to a web page for the product being advertised,a web page for an online merchant selling the product, the web page ofthe company which makes the product, or a notification of nearby storesdetermined to stock the product. If the media element is a trailer for amovie, the message may include a list of theaters and times where themovie will be shown, a hyperlink to a promotional web page for themovie, or a hyperlink to a third party web page or database withinformation about the movie. If the media element is a politicaladvertisement, the message may include a hyperlink to a candidate's orparty's web site, or an interactive message that allows the viewer toenter financial details for donation to a candidate or to sign up for acandidate's mailing list or consent to future communications from thecandidate.

FIG. 8 depicts another embodiment, wherein the viewer may be recorded bysensors while watching movie trailer 805 on display 800. The sensors maysend sensor readings to computing device 235, which may cross-referencethe sensor readings with records from the viewer's television provider,internet service provider, cellular network provider, or the like, andmay immediately determine that the viewer is demonstrating interest inmovie trailer 805. Computing device 235 may also determine the viewer'slocation, and cross-reference the movie in which the viewer demonstratesinterest with local theater offerings and times.

Computing device 235 may determine that the viewer is currently wearingor holding mobile display 810, and may cause message 815 to be displayedon the mobile display. The message 815 may inform the viewer thattickets to the movie advertised are available for purchase, and mayinclude a hyperlink to an online merchant page that sells tickets to themovie advertised. The message may also inform the viewer that there is acoupon or a special deal being offered to obtain the tickets at areduced price. Alternatively, computing device 235 may determine thatdisplay 800 is a device capable of displaying web pages, and may causemessage 815 to be displayed directly on display 800.

FIGS. 9a and 9b depict another embodiment, wherein the viewer may berecorded by sensors while watching media 905 on display 900. The sensorsmay send sensor readings to computing device 235, which maycross-reference the sensor readings with records from the viewer'stelevision provider, internet service provider, cellular networkprovider, or the like, and may immediately determine that the viewer isdemonstrating interest in media 905. The sensors may be used to thendetermine (for example, by accelerometer, motion sensor, globalpositioning system (GPS), or a camera) that the viewer has physicallymoved away from display 900. Computing device 235 may then cause amessage to be displayed on mobile device 910. The message may indicatethat the viewer may tap the message or otherwise input a command tocontinue watching media 905 on mobile device 910 while away from display900. Computing device 235 or another server may receive the viewer inputand cause mobile device 910 to display media 905.

The sensors may additionally be used to determine that the viewer hasreturned to display 900 and will be able to observe it again. Inresponse, computing device 235 may be configured to automatically causemobile device 910 to cease play media 905 and may automatically causedisplay 900 to continue to play media 905, so that the viewer has aseamless viewing experience regardless of location. Alternatively,computing device 235 may be configured to respond to determining thatthe viewer has returned to display 900 by sending a message offering toterminate media playback on mobile device 910, and not continue playbackon display 900 unless or until the viewer indicates via input responseto the message that the viewer wants to return to display 900

Alternatively, the viewer may begin watching media on mobile device 910,may be determined to have a positive reaction to the media, and then bedetermined to have approached display 900. Computing device 235 may beconfigured to respond by automatically displaying a message on mobiledevice 910 that the media may now be displayed on display 900 instead.In response to an input response from the viewer, computing device 235may then cause display 900 to begin playing the media at the same pointin time, and may cause mobile device 910 to cease playing the media.Alternatively, computing device 235 may be configured to automaticallycause the media to display on display 900 and to stop displaying onmobile device 910, in response to the viewer's pre-selection of anoption to automatically transfer media between viewing devices.

FIG. 10 depicts another embodiment, wherein the viewer may be recordedby sensors after entering a store and beginning to look at merchandise1000, such as a particular piece of clothing, offered for sale. Thesensors may be capable of determining the user's current location, forexample via GPS sensor or other location sensing system. The sensors mayadditionally be capable of sensing radio frequency identification (RFID)tags or other communication devices affixed to merchandise 1000, affixedto other merchandise, or placed throughout the store. The sensors maysend sensor readings to computing device 235, which may cross-referencethe sensor readings with maps of merchants, databases associatingparticular RFID tag or other tag information with merchants ormerchandise, or other information regarding merchants and merchandise.Computing device 235 may then be able to determine that the viewer isdemonstrating interest in merchandise 1000 or in the store generally.Computing device 235 may instead have previously determined an interestin merchandise 1000 due to the viewer's reaction to media featuringmerchandise 1000 or other brands on media consumed by the viewer in theviewer's home.

Computing device 235 may be configured to respond to this determinationby causing a message 1010 to display on mobile device 1005. Message 1010may offer a coupon to purchase merchandise 1000 at a reduced price, mayalert the user that the merchandise is available only for a limitedtime, may offer a coupon for merchandise at the store generally, mayinform the viewer of currently-offered deals that require the purchaseof merchandise 1000 as a prerequisite to receive another benefit or thatresult in receiving merchandise 1000 for free or for a reduced price, ormay inform the viewer of all currently-offered deals at the store.

Computing device 235 may additionally cause the display of or offerdisplay of other relevant information to the viewer in response to thedetermination of the viewer's location and interest. Computing device235 may cause display of hyperlinks to web pages of the manufacturer ordesigner of merchandise 1000 or other relevant merchandise, hyperlinksto web pages with reviews of merchandise 1000 or other relevantmerchandise, or hyperlinks to an online merchant who also sellsmerchandise 1000 or other relevant merchandise.

Instead of or additionally to causing display of a message on mobiledevice 1005 while the viewer is present in the store, computing device235 may update the viewer's emotional response profile and may at afuture time display an offer to sell merchandise 1000 to the viewer,remind the viewer that merchandise 1000 is still available for sale, orlink the viewer to an online merchant who sells merchandise 1000.

In another embodiment, media may be offered for sale or rent to aviewer. The viewer may be recorded by sensors while watching a programadvertisement on a display. The sensors may send the readings tocomputing device 235, which may cross-reference the sensor data withrecords from the viewer's television provider, and may immediatelydetermine that the viewer is experiencing a positive reaction to theadvertisement, such as excitement or happiness. Computing device 235 maydetermine that the viewer is capable of purchasing the rights to viewthe program being advertised, such as through pay-per-view, video ondemand, rental through a third party, or a paid video download.Computing device 235 may then cause a display associated with the viewerto alert the viewer that purchase, rental, or licensing of the programis available, and may include a link to an online merchant to which theviewer may go in order to do so.

In another embodiment, targeted advertising may be improved. Computingdevice 235 may transmit previously determined information regarding aviewer's emotional responses to particular media elements, or even aviewer's complete emotional response profile, to another computingdevice, such as an advertising server. The advertising server may thenuse the transmitted information to determine what advertisements toinsert in media later viewed by the viewer. The advertising server mayinsert advertisements into web pages, into mobile applications, intostreaming video, within television programs, during commercial breaksbetween parts of television programs, or in any other medium that may beobserved by a viewer. The insertion may instead be performed by mediaserver in response to instructions from the advertising server.

The advertising server may determine which advertisement to insert basedon an advertisement containing elements which have been determined toelicit a positive reaction from the viewer, based on an advertisementcontaining elements similar to those determined to elicit a positivereaction from the viewer in the past, or based on an advertisement notcontaining elements which have been determined to elicit a negativereaction in the past.

The advertising server may also display a message to the viewer beforedisplaying any advertisements, offering the viewer a choice between twoor more advertisements which are determined to be likely to be viewedpositively by the viewer. The viewer may select one of theadvertisements to view by clicking, pressing a button, or otherwiseindicating input which is relayed to the advertising server. Theadvertisement server may then cause the selected advertisement to beshown to the viewer. Alternatively, the advertising server may displayone advertisement by default, but may include a message to the viewerthat a different advertisement is available for viewing. The viewer mayclick, press a button, or communicate other input to the advertisementserver to cause the second advertisement to be displayed instead.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

I claim:
 1. A method, comprising: determining a change in one or morephysical attributes of a viewer over a time period during which a firstmedia element is displayed to the viewer; determining an emotionalresponse of the viewer to the first media element, based at least inpart on the change in one or more physical attributes; selecting asecond media element based at least in part on the emotional response ofthe viewer to the first media element; and transmitting the second mediaelement to the viewer.
 2. The method of claim 1, wherein the one or morephysical attributes of the viewer over the time period are detected viasensors of a wearable device.
 3. The method of claim 1, wherein thesecond media element comprises an offer to sell a good or service whichwas displayed in the first media element.
 4. The method of claim 1,further comprising: prior to transmitting the second media element tothe viewer, offering to record content that the viewer can request forreplay at a later date.
 5. The method of claim 1, further comprising:prior to transmitting the second media element to the viewer, notifyingthe viewer that the second media element is currently available from amedia source, and offering to display the media source.
 6. The method ofclaim 1, further comprising: prior to transmitting the second mediaelement to the viewer, notifying the viewer that media can be displayedon-demand to the viewer.
 7. The method of claim 1, wherein the firstmedia element is displayed to the viewer on a first display, and thesecond media element is displayed on a second display different from thefirst display.
 8. The method of claim 1, further comprising: prior totransmitting the second media element to the viewer, notifying theviewer that the second media element is available for purchase orlicense.
 9. The method of claim 1, wherein the determining an emotionalresponse of the viewer to the first media element comprises generatingor amending a viewer interest profile.
 10. The method of claim 1,wherein the selecting a second media element to be displayed to theviewer is additionally based at least in part on a viewer's determinedlocation.
 11. A method implemented by one or more computing devices,comprising: receiving, from a wearable sensor device, informationrelated to one or more sensor readings associated with a viewer,receiving an indication of a first media element the viewer viewed at atime when the one or more sensor readings were taken, determining anemotional response of the viewer to the first media element based atleast on said information, selecting a second media element based atleast in part on the emotional response of the viewer to the first mediaelement; and offering to present the second media element to the viewer.12. The method of claim 11, wherein the one or more sensor readingsassociated with a viewer comprise sensor readings of an environmentalcharacteristic.
 13. The method of claim 11, wherein the second mediaelement comprises an offer to sell a product.
 14. The method of claim11, wherein the offering to present the second media element to theviewer comprises an offer to record a streaming media that the viewercan request for replay at a later date.
 15. The method of claim 11,wherein the second media element is further selected based at least inpart on a location where the viewer is determined to be.
 16. Anapparatus comprising: a mobile display, and a processor configured to:receive, from a sensor device, information related to one or more sensorreadings associated with a viewer, receive an indication of a firstmedia element the viewer viewed at a time when the one or more sensorreadings were taken, determine an emotional response of the viewer tothe first media element based at least on said information, select asecond media element based at least in part on the emotional response ofthe viewer to the first media element; and offer to present the secondmedia element to the viewer.
 17. The apparatus of claim 16, wherein thefirst media element is displayed on a second display, and the offer topresent the second media element comprises an offer to view the secondmedia element on the mobile display.
 18. The apparatus of claim 17,wherein the first media element and second media element are bothportions of a same video content.
 19. The apparatus of claim 16, whereinthe second media element comprises information about the first mediaelement.
 20. The apparatus of claim 16, wherein the second media elementcomprises an offer to sell the first media element or an item displayedin the first media element to the viewer.